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ABSTRACT 

Evaluation research is currently facing a crisis. At 
the same time that federal and other funding sources are placing 
increasing emphasis on evaluating on-going programs, action personnel 
are becoming more and more skeptical about the value of such 
research. Evidence of this skepticism can be found at any gathering 
of evaluators. Certainly, different frames of reference guide 
researchers and educational programers. However, in the long run, 
both are aiming at the same goal — a more effective program. This 
suggests that there is enough common ground between researchers and 
programers to provide a basis for working together, and that some 
adequate compromise can be struck between the needs of research and 
those of education programs., (author/mf) 
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Research or Subjeotave Judgment 

by 



Eleanor K* Caplan, Fa.D. 



Evalu‘’aion I'esoarch is 



currently facing a crisis* 



At the same 



time that Federal and other funding sources are placing increasing 



emphasis on evaluating on»-going programs and demonstrations, action 



personnel are becom5.ng more and more skeptical about the value of 



such research. Evidence of this can be found at any gathering of 
ovaluaoion rese^archers. At such meetings, the conversation inevitably 



turns to the '‘inability” of action personnel to understand the needs and 



requirements of research, or to appreciate what research can do for them. 



Different frmaes of reference guide researchers and educational 



programirflers. However, in the long lun, both are aiming at the same 



4 . 

goal — — a more effective program. This suggests that there is enough 



common ground between researchers and the-directors and staff of edu- 



cation programs to provide a basis for working together, and that 



some adequate compromise can be struck between the needs of research 



and those of education programs. Outlining some of the steps necessary 
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Exrporience suggests that such a comprowis© must begin >rith a 



rethinking of the role of research by those engaged in it^ Rethinking 

of the role of ©valuation research in educational program’s requires 

consideration of definitions of such research — obtaining a clear 

understanding of >?hat ©valuation research is, and vfhat distinguishes 

it from, other kinds of research. 

Although evaluation research resembles other social scientific 

research in its tools and methodology, it differs from such research 

limited 

in two major features: Its purpose, .and the/control which the researcher 

has over the conditions and structure of the research. 

For all programs designed to bring about change, evaluation 
has been defined as the determination of the results attained by ' 
some activity designed to accomplish some valued goal or objective. 

Adding the word research to the basic concept of evaluation implies 
the use of the scientific method to obtain this infonnation. 

In education, ©valuation research is essentially a pi’ocess of 
making a rational, documented judgment about the effectiveness of a 

course in achieving the ends set for it by its designers. As such, 

!• Suchman, Edx^ard P, Evaluative Research; Principles and Practice 

in Public Service and Social Action Programs . New York; Russell 
Sage Foundation, ~ 1967 . Pp. 31-32, 
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its purpose is to pxxjvido educatox's with feedback concerning the success 
or failux’c of their cffoi'ts> and the reasons for it. 

This information is intended to enable educational piogrammcrs 
to iraprove on-going and future courses and curxdcula, and to persuade 

“taxpayers and parents that adequate results are coming form the billions 

2 . 

spent for education each year.” It is apparent, then, that evalua- 
tion is a peculiar species of resoarch, in that its major function is 
to pix>vide sorxdce to prograjtis and funding sources. In this it differs 



from basic research in which answers are sought for their ovm sake. 
Failure on the part of researchers to keep this in mind can compound the 
already complex problems of evaluation research. 

Researchers &re draim into en educat5.onal situation to “be parti- 
cipants in the policy-making process through the exercise of special 

3 . 

skills.” They are thus as intimately involved in and responsible for 



the prograra as its staff membei's, and should be equally concerned with 
its ultimate success. Forcing themselvor to accaept this, and to 



abandon the outlook of the outside researcher passing some sort of 

2. Iwarx)to, DavidV and xJoii:tiarrE'r Hoarn^^ “Bvaluation is a Full-Time 
Job,” Meric an Sd|Z cation . April, 1969* 

3* Rossi, Peter H. Practice, Method and Tlieory in Evaluatlno: Social 
Action Prop:rnjns . Revised version of a paper presented at the 
annual meeting of the American Statistical ASwSociation, 1966. 
March, 1968. 



Olympian judgin©nt on a course is a vital first step in establdshing 
a productive vrorking relationship tvlth program staff. 

Evaluation research also differs from basic I'esoarch in the ardount 

of control \ji:iich the researcher has over the conduct and conditions of the 

0 

research. The research is in and of something larger than itself — — 
the program ——and, because it is apx>lied research in the strictest 
sense of the term, its goals are subordinate to those of the larger 
entity. 

In this situation, the researcher retains control and 

responsibility for the choice of conceptual framework and for the 
technical aspects of collecting and analysing the data. Ho remains 
dependent on the program director, hovrover, in several important 
matters. The programmer is in control of program, personnel, and 
eveiy activity v^iich occurs T-dthin the context of the program. Thus 
the researcher is dependent upon the good will of the program director 
at nearly every stop of the research process. The program director, 
on the other hand, imist view the needs and goals of research as secon- 
da.ry to those of tlie program as a whole, and so frequently looks upon 
the precence of the researcher as a necessary evil at best. 
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It is imperative that the researcher understand these two facts 
about evaluatioii research that its role in a program is instiai- 
mental and suppoi*tive, rather than primary, and that it is su1x>rdinate 
to and d.ependent upon the program and its staff* Once this understanding 
is established, the researcher is prepared to take steps to maximize 
a mutually productive relationship between himsedf and the staff and 
directors of the program# 

The first responsibility of the researcher is to make the role 
of research in the program, and its potential value, clear to the 
program director and his staff. Many reseaichers fail to realize 
initially that most action personnel are skeptical about the value of 
research • 

This skepticism stems from a number of causes — - some rational 
and some purely subject!^;© and personal. ^Vny times, evaluation re- 
search has been forced on the program by fiat from a funding source; 
resistance is such cases is understandable. Frequently, the necessarily 
ru.thless analysis of programi objectives and activities is viewed as an 
effort to undermine the program and question the competence of the staff. 



o 
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Research activities invariably intorforo vdth the ongoing work of the 
program to sorae degree, and consume a portion of tho total program! 
budget which ormnitted pjx>grammers feel shorld be used in direct assis- 
tance to tho tar»get population. 

The dependence of tho researcher on the piograra personnel moans 
that :< understand5.ng of, and commitment to, the research on 

the part of the progranmiers are essential to tho ultimate success of 
the research effort. Unless this ind-tial skepticism, and the z'osis- 
tance which springs from it, are oversome early in tho relationship, 
any hope of a productive collaboration between research and pz'ogram 
personnel is lost. For this reason, the I'esearchers should feel a 
strong obligation to stabilize and clar3.,fy the relationship between tho 
research and the pi*ogram before the particular design for the research 
is chosen. 

The second major step -which the researcher must undertake is the 
establishment and maintenance of tho ongoing and cooperative interaction 
with -the program director and professional stgJTf which -vriLll make it 



possible for the research to fulfill its role 



Tho initiation end early stages of this interaction must coincide 



>dth the preplanning and planning phases of both program and research. 
Research vjhich is initiated and planned after the program begins 
operations con never fulfill the potential ;diich both tho resaercher 
and tho progrcamer should expect from it; raany effective research tools 
and designs are elir/Linated from consideration by such a late start, 

And> just as tho researcher expects a.nd needs to be included in 
the planning of the courses included in the program, so must he recog- 
nize tho id.ght and need of the prograra director to be included in the 
planning of research. The phases of pre-evaluation planning in "which 
interaction between research and program staff is vital include; The 
statement of the problem, formulation of hypotheses, and selection of 
a research design. 

In evaluation research, the problem and the hypotheses \diich 
procode from it are to a large degree pre-dtermlned. An educational 
demonstration or other action progi'ar/i is in3.tiated in resix>nse to an 
identified problem or deficiency in tho community. Furthermore, the 



form and nature of the program is based upon an overt or covert hyjx)- 



MIP 
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thesis about the cause of the p'x*oblem and the Idnds of actions vjhich 
vrl3,l allerlate it* 

Evaluation I’eseai'ch is essentially designed to dotemlne i^;hother * 
these actions do in fact alleviate the problem p in i;hat degree, and 
Under ijhat circurastancos. Thus, a clear statement, by and with 

progi'a^iners, of the pr-oblem under attack, of the assumed relationship 

* 

between it and the pr-ogranmiatic solutions, and the expected or desired 
consequences of the action of pi'ogram activities on the problem are 
essential to effective development’ of a research design* 

Contrary to the opinion of many pi'^gram personnel, the researchers 
are not better equipped or able to p 3 X)vide this kind of statement than 
the program director and his staff* He can, however, assi^st the program 
personnel in making’-the nature of the problem, hypotheses and expected 
results explicit enough to fom the basis for design of a pertinent 
methodology* 

^en this kind of cooperation occurs early in the program planning 
phase, it frequently provides a valuable by--product in better under- 
standing on the part of prograr.miers of precisely what they want to do. 

o 

ERIC 





This can make it possible for the programmer to include in the design 
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of the progx*am activities roor© closely alligned with its ultimate 
purpose(s). 

Cooperation between researchers and px’ogram director's and staff 
is also essential in selection of the basic research design. IJany 
researchers — — and programmers — feel that| once program goals and 
objectives are adequately specified, selection of ^jn appropriate design 
follows naturally, and can be done semi-autoraatically by tho researcher. 

I propose that this is r*ot the case, and that; further, this 
assxim.ption on the part of the researcher is a heavy contributor to 
subsequent friction and misunderstandirg between researcher and prograia- 13 

rners* 

T^o possible variations in ovaliiation research design are even 
great©!' than in non-evaluative research — runningthe gamut from sub- 
jective judgment to the most rigorous experiment. A brief considera- 
tion of some of the forms idiich such designs can take may help to illustrate 
this. In his book on evaluation research, Suchman provide^s a brief 
inventory of evaluative research designs. The ones most commonly 

used in evaluation studies are; 

51 Ibid. , pp^91"-1i5. 
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The Fre^Tes t, Post>/rest,..qoMrpJ.^l ^^^ 



setting up two eqiii valent gr-oups of e\ibjectS| eithea." 



This consists of 
by matching of 



random assignment# Both groups are then adm3,n3.sXf6red some soru of 
instiiiiiient designed to establish a oase line from Tfjhich change —~— 
hopefully due to the intervention of the program - — can be evaluated. 
One group is then r^xposed to the prograpij and the other is not# The 
measuring instn^nent is acbninistercd a second time, and the before and 
after scores of each group are corapai'‘ed# The test of program effec*~ 
tiveness is the difference between the scores of the group which was 



eucposed to the program and the one which was not# 



The use of the control group, and of before and after measurements, 



is intended to insure that change has occured, and that this change 



can be attributed to the intervention of the program# 

This is the classic experimental design, translated from the tra- 
ditional laboratory setting to the *'field#” All other designs in 
social science research which can legitimately be called research are 

“ variations on this design primaidlly compromiser, necessitated by 

limited funds or force of circumstances# It must be clearly understood 
that each compromise mth the rigorous structure of this *‘ideal»» design 



lessens the reliability and explanatoiy povrer of the results* 



The Static Group Comparison is simlar to the ideal design in that 
twp groups are usodj and that one is exposed to the pi\3grara and one is 
not. However, the jneasuring instrument is admdnistered only after 
e:q?osjur 0 , and the test of program effectiveness is the difference 
between the scores of tho two gi’oups on this post-tost. The principal 
problem with this design is that one does not know vjhethor the two 
group»s would have had equivalent scores on the instrument if they had 
been tested before progx'sm exposure. 

The One Group t Pre-Tovst^ Post-Test Desip’n . In this design, only 
one group is used — — the subjects «nroHed in tho program -—and they 
are administered tho measuring instrument before and after program 
exposure. Tho test of program effectiveness is the difference between 
tho scores of the group in tho before and after measurements. The 
weakness of this design is that change as measuerd may not bo the 
result of participation in tho program, but of some extraneous influence 
in the lives of tho subjects. 

The One-Shot Case Study . This design uses only the participant 



grou]:»,. and admirdotorrs the meamiring Instrument only after program 



cjqposure. There ax'e several vai'detics of one-shot design* One 
frequently used is vjhat Suchnan apt3.y cal!l.s the ’*is everybody happy” 
design, vrhich tal<es testimoidals from the participants# A3.thoufih 



this frequently gives insights into the effectiveness of the program, 



the results have little scientific validity# 

A more ”objeotlve” vaidation on the one-shot design adiuinisters 
instruments designed to obtain more of a,TOsasure of change from the 
subjects than their opinions of the program# However, even in this 
case, one cannot assess the amount of change, nor safely attribute it 
to the intervention of the program# 

The I-onfdtudinal Design resembles the One-Shot Case Study in that 
it uses one group# However, the gix>up is administered the measuring 
instrument periodically over a long time span, and theii* pl^^gross is* 
charted from comparisons of scores over time# Althou^ problems of 
attributing change to the program still exists, this is a valuable 
tool for evaluating long-term, ongoing programs, vjhere cross-vsectional 



designs rrd^t provide distorted or inadequte results# 
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Ii ono can say that theso dej^.gns range theraselves on a continuuni;, 
VTith the most powerful and rigoi’ous design - — the classic experiment — — 
at one ond^ then the last design to be considered must certairay fall 
at the other end. TliiwS design Suchman calls ’’evaluation” rather than 
©valuatD,on research j on the grounds that^ although it too involves 
making a juclgruent of worth or effectivenes5;| it includes no ’'systematic 

procedures for marshalling anf presenting objective (scientific) evidence 

5 . 

to support the judgraent.” This is the process frequently used by 

program directors and staff in irjhich the year«“©nd evaluation consists 

of a narrative recording their very professional, but veiy unsystematic 

impressions of the progress of the pi’ogram. 

In evaluation, as in other kinds of research, the decision as to 

idilch of these designs to xise must be based on a number of factors. 

Some of these are purely research considerations , and a number are 

matters of administration, finance, and policy concerns. The latter 

are, and must be recognised as the pi’ovince of the pi’ogram director. 

The educational programtier is operating in a situation in which 

ho must obtain the maximum value fi*om increasingly scarce resources. 
Ibid., p. 7. ~ 






Tlie best and niost rXgorously scientific research may not be best for 
the program; it may eat up too large a proportion of the allocated 
funds, it may be too sophisticated for the purposes of the program, 
or it may be too disn,iptive of program operations. And, the researcher 
must keep in mind that, in the long run, is best for the program 
is a docision \diich must and >rlll be made by the progrrum director. 

The researcher has, however, a particular responsibility in this 
situation. Because of his specialised knowledge of the ins-and-outs 
of research, the reseai*cher must make the vaidous feasible alternatives 
in design, and their costs and implications, known to the program director. 

Thus, successful pursuance of evaluation research miASt be built on 
close cooperation and an apparopriate d3.vision of labor between research 
and program pej'sonnel. Both must recognise and act upon their mutual 
depondenco, while recognising and ejqploiting the competencies of each, 
Hoprefully, the result can be evaluation research which is meaningful 



both in sole ttific and action contexts. 



OPTIONS FOR EVALUATION OF AOJLT BIJUCATION: 
Research or Subjective Judgment 



Eleanor K* Caplan, Ph 
SUMMARY 




Evaluation research is currently facing a crisis. At the same 
time that Federal and other funding sources are placing increasing 
emphasis on evaluating on-going programs and demonstrations, action 
personnel are becoming more and more skeptical about the value of 
such research. 

Diffei*ent frames of reference guide researchers and educational 
programmers. However, in the long run, both are aiming at the 
same goal — - a more effective program. This suggests that there 
is enough common ground between researchers and the directors and 
staff of education programs to provide a basis for working together, 
and that some adeqaute compromise can be struck between the needs 
of research and those of education programs. Outlining some of the 
steps necessary to reaching this compromise is the purpose of this 
paper. 

First, the researcher must understand and come to terras with 
the fact that evaluation research differs from non-e'v^uative, or 
basic, research in its purposes, and in the limited control \diich 
the researcher has over, the research situation. 

Secondly, he must take the initiative in making the role, and 
the potential value of research^to the educational program. Initial 
resistance to research and researchers, Aether it is based on rational 
or purely personal and subjective causes, must be overtorae if the . 
research is to be of any value to the program. 

Thirdly, the researcher must endeavor to establish and maintain 



an on-going, ocoperativo interaction with the program director and 
staff. The initiation and early stages of this interaction must 
precede both the commencement of the program and the choice of a 
research design. 

This interaction must, in these early stages, particularly 
focus on three elements ofpre-evaluation planning: Statement of the 
poblem, formulation of hypotheses, and selection of a research design. 
Since the problem and research hypotheses spring from the goals and 
stimuli of the program, the program director’s particular knowledge 
of the program demands that he be included in the discussions which 
culminate in decisions in these matters. 

Moreover, the ultimate decision about the choice of research 

design is the prerogative of the program director. The design, 

primarily 

and the research itself, have value p)(%f/in their contribution to 
the improvement of the program and the amelioration of the problem 
to which it is addressed. 

The educational programmer is operating in a situation in \diioh 
he must obtain the maximum value from increasingly scarce resources. 
The best and most rigorously scientific research may not be best for 
the program | it may eat up too large a proportion of the allocated 
funds, or it may be too sophisticated for the problem to whiah it is 
addressed. And, the researcher must keep in mind that, in the illong 
run, what is best for the program is a decision which must and will 
bo made by the program director. 

The researcher has, however, a particular responsibility in this 
situation. ^he variations in design of evaluative researchare even 



great©!? than In non-evaluativ© research — — running the gaKcut from 
subjective judgment to the most rigorous experiment. Because of his 
specialized knowledge of the ins-and-outs of research, the researcher 
must make the various feasible alteiviatives in design, and their 
costs and implications, known to the program director. 

Thus, successful pursuance of ©valuation research must be built 
on close cooperation and an appropriate division of labor between 
researdbi and program personnel* Both must recognize and act upon 
their mutual dependence, while recognizing and exploiting the com- 
petencies of each. Hopefully, the result can be evaluation research 
which is meaningful both in scientific and action contexts. 



ERIC Cleurioghouse 



FEB 2 5 1970 

on Atiuut iiduakdoB 



